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The present invention is directed to a method according to the 
definition of the species in Claim 1 and to a device suited 
for carrying out the method according to the definition of the 
species in Claim 15. 

By detecting an imminent collision between a road user and the 
vehicle of an observer at an early stage, one can improve the 
safety of vehicle occupants as well as of the potential other 
party involved in the collision. The time savings that can be 
gained by visually detecting and evaluating the observable 
region in front of the observer's vehicle permits stepped 
reactions of the vehicle occupant safety systems (e.g., a 
gentle firing of the airbag) , or makes reactions possible in 
the first place to protect the other party in a collision 

(such as raising the engine hood in the case of collision 
involving a pedestrian) . Developments culminating in 
switchable crash structures have made it more important than 
ever to know the type of other party involved in the accident 

(truck, passenger car, motor cycle, pedestrian) . 

Current developments in precrash sensors are directed, inter 
alia, to the analysis of methods based on infrared lasers, 
ultrasound, or radar sensors. The drawbacks of these systems 
are due, in part, to their limited range (ultrasound, infrared 
laser) and to their inability to identify the other party 
potentially involved in a collision, along the lines of a 
reliable type classification (truck, passenger car, motor 
cycle, person) . In radar-based systems, inter alia, non- 
metallic objects (e.g.: people or trees) are not able to be 



reliably detected by inexpensive sensors suited for use in 
vehicle applications. However, adapting an optimally stepped 
reaction of safety systems to imminent collisions requires 
reliable detection and dependable typing. Within the framework 
5 of such a stepped reaction, for example in the event of a 

collision with a pedestrian, active measures should be taken 
to ensure the pedestrian's safety. In this connection, one can 
conceive of rapid changes in the vehicle body form to minimize 
the probability of serious head or leg injuries. However, the 
10 basic condition for activating these measures is that the 
safety system be able to reliably detect road users and 
classify them according to type (e.g.: passenger car, truck, 
bicycle riders, pedestrians) . 

15 Generally, methods for interpreting image scenes endeavor to 
not only obtain purely two-dimensional image information, but 
multi-dimensional scene information as well. They do so 
already in the first step, using a mostly costly sensor 
(stereo sensor or high-resolution radar or lidar) . In order to 

20 detect objects, these methods are based, however, on models, 
above all with respect to the position and orientation of 
potential targets, as well as with respect to a predefined, 
fixed geometry of the orientation to the sensor and 
surroundings. In practice, it is often ascertained, however, 

25 that these models and assumptions frequently do not conform 
with actual conditions, so that misinterpretations result. 

The object of the present invention is to devise a method and 
a suitable device which will make it possible to detect road 
30 users on the basis of camera images, to determine their 

distance from the observer, and to classify the road users. 

The objective is achieved by a method and by a device suited 
for implementing this method and having the features described 
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in Claims 1 and 15. In this case, the recording of data, 
organized into a plurality of steps, and object identification 
facilitate the use of conventional sensory technology and, at 
the same time, offer the potential of a real-time 
5 implementation. 

Advantageous embodiments and further refinements of the 
present invention are indicated in the subordinate claims. 

10 The method according to the present invention identifies 
regions within a camera image, in which road users or 
obstacles are situated. It advantageously suffices, in this 
context, that this camera image contains purely two- 
dimensional image information without any distance resolution. 

15 The identification is carried out using a classifier specially 
trained for the road users and obstacles to be detected. In a 
subsequent step, the regions so identified are then marked and 
ranged using a distance-measuring sensor with respect to their 
distance from the observer. Selected regions are subsequently 

20 fed to a type classification to precisely determine the type 
of road user or obstacle. 

A device suited for implementing this method includes a mono- 
image camera, which is coupled to a distance-measuring sensor 
25 unit. In each case, this coupling has an interposed or 
downstream classifying unit. 

In one advantageous embodiment of the device, the downstream 
classifying unit used for classifying types has an upstream 
30 selection unit connected in incoming circuit thereto, with 

whose assistance the number of regions to be classified can be 
controlled . 
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Figure 1 schematically depicts one advantageous refinement of 
such a device for implementing the method of the present 
invention. Here, mono-image camera 1 makes image data 10 
available to a classifying unit 2, which identifies image 
5 regions containing the road users or obstacles and informs 
distance-measuring sensor unit 3 of the corresponding 
positional data. Sensor unit 3 then measures these regions 
with respect to their distance from the observer. In so doing, 
these measuring data 30, together with data 20 from first 

10 classifying unit 2, are available to a selection unit 4. 

Selection unit 4 can control the data flow to the downstream 
unit for classifying selected road users or obstacles 5 by 
type. Image data 40 selected by the selection unit are 
transmitted to classifying unit 5 for classification. Results 

15 50 of this type classification are advantageously made 

available to a risk calculator connected to the classifying 
unit, so that the risk calculator can decide whether to 
initiate reactions that meet the requirements of the 
situation. 

20 

It is also conceivable, in another advantageous embodiment of 
the device according to the present invention, for selection 
unit 4 to be omitted and, basically, for all data 20 and 30 to 
be fed directly to classifying unit 5. 

25 

The method according to the present invention can be devised 
quite advantageously to identify road users and obstacles 
through the use of a hyperpermutation network within the 
framework of classifying unit 2. 

30 

A network of this kind is able to localize regions belonging 
to a specific class (in this case road users and obstacles) , 
at a high speed and on the basis of pixels, within image data 
10. The advantage of using simple, two-dimensional image 
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information is not only apparent in the method' s suitability 
for using a simple, inexpensive camera, but, in particular, 
also in the feasibility of using powerful classification 
algorithms under real-time conditions to analyze the entire 
5 image information. Since image information 10 supplied by a 
mono-image camera 1 is quite simple, it is possible, in 
contrast to conventional methods that mostly work on very 
complex data, to include every single image pixel in the 
classification . 

10 

Within classifying unit 2, an algorithm, advantageously a box 
(boxcar-averaging) algorithm adapted to this task follows the 
actual classifier (for example the hyperpermutation network) . 
This box algorithm combines and marks related regions of 
15 interest ROI, so that they can be fed for further processing. 

Within the framework of this further processing, the regions 
belonging to this ROI are marked and ranged using a distance- 
measuring sensor 3, with respect to their distance from the 

20 observer. Radar systems or stereo camera systems are 

advantageously suited for this. Since the data acquired by 
these sensors are utilized purely for estimating distances and 
not for classifying types of road users or obstacles, there is 
no need to equip these sensors with extreme angular 

25 resolutions or with robust models requiring substantial 

computational outlay. Thus, one can preferably revert to using 
sensors which are already present in the vehicle and mainly 
geared towards other applications. 

30 It is also conceivable, however, in another advantageous 

embodiment of the method according to the present invention, 
for distances to be estimated in the area of the ROI using a 
mono-image camera, in cooperation with a complex image 
analysis. Since, in this connection, only individual segments 
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(ROI) of the entire image information need to be processed, 
large-volume computational work can be performed in real time 
using powerful processors. It would be especially beneficial, 
in this context, if this information could be directly 
5 obtained in a second processing step from image data 10 

already supplied by mono-image camera 1. The system can also 
be implemented without the use of an additional distance- 
measuring sensor 3. 

10 In the method of the present invention, the information 

obtained by repeatedly measuring the distance of road users or 
obstacles from the user is advantageously used to determine 
the relative velocity of these objects in relation to the 
observer. It is of particular benefit for distance-measuring 

15 sensor 3 to not only provide distance information, but 

velocity information as well (e.g.: double radar). This would 
enable the indirect velocity estimation to be omitted from the 
sequence of distance measurements. 

20 The distance and velocity information 30 is fed, together with 
image information 10, to a selection unit 4. Based on its 
default settings, this selection unit then decides which of 
the image data are to be supplied to a type classification 
within a downstream classifying unit 5. One can conceive of 

25 selection unit 4 being configured to basically supply all 

image data belonging to ROI to a type classification. However, 
one can also conceive of only those image data being 
transmitted which belong to the ROI having assigned road users 
or obstacles which meet specific criteria. In the process, the 

30 existing potential for danger is to be considered; thus, for 
example, the size of objects or the velocity at which they 
move toward the observer, or also their relative velocity in 
general (for example, moving or stationary objects) . 
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For the actual type classification within classifying unit 5 
which serves the purpose of precisely defining the type of 
road user or obstacle, one can fall back on classification 
algorithms specially trained for such objects. Advantageously 
suited for this are neural networks, such as a radial-basis 
function classifier or a support-vector machine. 

The method and device of the present invention are superbly 
suited for the early detection and subsequent assessment of 
accident situations (precrash detection) . 



What is claimed is: 



1. A method for detecting road users and obstacles on the 
basis of camera images, in order to determine their distance 
from the observer and to classify them, 

wherein 

a classifier designed for detecting road users and obstacles 
identifies regions within a two-dimensional camera image that 
is not resolved with respect to distance; 

in a subsequent step, these regions, so identified, are marked 
and then ranged using a distance-measuring sensor with respect 
to their distance from the observer; 

and selected regions are subsequently fed to a type 
classification to identify the road users or obstacles. 

2. The method as recited in Claim 1, 

wherein the camera image, which is utilized for identifying 
road users, contains only two-dimensional image information 
without any distance resolution. 

3. The method as recited in one of Claims 1 or 2, 

wherein the information obtained from the measurement is used 
to determine the relative velocity of the individual road 
users or obstacles . 

4. The method as recited in one of Claims 1 through 3, 
wherein the classifier designed for recognizing road users is 
a hyperpermutation network. 

5. The method as recited in one of Claims 1 through. 4, 
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wherein a box algorithm is used to mark regions identified as 
road users. 

6. The method as recited in one of Claims 1 through 5, 
wherein the distance-measuring sensor is a radar sensor. 

7. The method as recited in one of Claims 1 through 5, 
wherein the distance-measuring sensor is a stereo-camera 
system. 

8. The method as recited in one of Claims 1 through 5, 
wherein the distance-measuring sensor is a mono-camera system, 
which, by using suitable image processing, is able to make 
distance estimates . 

9. The method as recited in Claim 8, 

wherein the mono-camera system is the same system used already 
in the first step to generate the two-dimensional camera image 
that is not resolved with respect to distance. 

10. The method as recited in one of Claims 1 through 9, 
wherein the classifier used for type classification is a 
radial-basis function classifier. 

11. The method as recited in one of Claims 1 through 9, 
wherein the classifier used for type classification is a 
support vector machine. 

12. The method as recited in one of Claims 1 through 11, 
wherein the regions to be subjected to a type classification 
are selected as a function of their distance and/or relative 
velocity in relation to the observer. 

13. The method as recited in one of Claims 1 through 11, 
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wherein the selection of the regions to be subjected to a type 
classification includes all regions identified as road users 
or obstacles . 

14. The method as recited in one of the preceding claims, 
wherein the result of the type classification is transmitted 
to a risk calculator to decide on reactions to be possibly 
initiated. 

15. A device for detecting road users and obstacles on the 
basis of camera images, to determine their distance from the 
observer, and to classify them, 

wherein 

a mono-image camera 1 is coupled to a distance-measuring 
sensor unit 3; 

and, in each case, this coupling has an interposed or 
downstream classifying unit 2 and 5, respectively. 

16. The device as recited in Claim 15, 
wherein 

the mono-image camera 1 is linked to a classifying unit 2; 

this classifying unit 2 contains a module for identifying 
image regions to be assigned to road users and obstacles, and 
which marks these regions and makes available the 
corresponding data 2 0 at the output of the module for a 
further processing; 

coupled to the output of the module is a distance-measuring 
sensor unit 3, which is able to measure the marked regions 
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with respect to their distance from the observer, and which 
makes available these measured data via a connection 30 to a 
selection unit 4, via which a second classifying unit 5 is 
linked to the entire system and classifies the regions 40 
supplied to it by the selection unit with respect to type of 
road user or obstacle. 

17. The device as recited in one of Claims 15 or 16, 
wherein a risk calculator is connected to the output of the 
classifying unit for type classification 5. 

18. A use of the method as recited in one of Claims 1 through 
14 for early detection of accident situations (precrash 
detection) . 

19. A use of the device as recited in one of Claims 15 through 
17 for early detection of accident situations (precrash 
detection) . 
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Abstract 

A method and a device for detecting road users and obstacles 
on the basis of camera images, in order to determine their 
distance from the observer and to classify them. In a two-step 
classification, potential other parties involved in a 
collision are detected and identified. In so doing, in a first 
step, potential other parties involved in a collision are 
marked in the image data of a mono-image camera; their 
distance and relative velocity are subsequently determined so 
that endangering objects can be selectively subjected to a 
type classification in real time. By breaking down the 
detection activity into a plurality of steps, the real-time 
capability of the system is also rendered possible using 
conventional sensors already present in the vehicle. 
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